Skip to content

Fixture proof: one phase's context recipe, byte-identical across eight passes (LAB-136) - #1325

Merged
jhoffner merged 11 commits into
mainfrom
fix/LAB-136
Aug 19, 2026
Merged

jhoffner merged 11 commits into
mainfrom
fix/LAB-136

Conversation

@jhoffner

@jhoffner jhoffner commented Aug 18, 2026 •

Copy link
Copy Markdown
Collaborator

The problem

We tell people that a phase of work gets exactly the context it needs, assembled the same way on every pass. Nothing tested that. If the assembly quietly varies between passes — a timestamp here, a differently-ordered list there — then round eight does not behave like round one, and the reliability the whole system is being built for is not there.

Why now. The two layers that would make the assembly observable are deliberately last in the epic's build order. On that plan the claim stays untested until everything sits on top of it. Proving it here costs one goal check. Discovering it after the substrate is built costs the substrate.

What this does

Hand-builds a small world as literal fixtures — two issues, their pull requests, the diffs, the objective they sit under — with no database and no event feed. Builds one phase's context recipe against it (review), then checks two things: that a person reading the resulting brief agrees it is what a reviewer needs, and that the machine gets the same bytes eight times running.

The brief is shaped as a standing half and a small changing half. Only the bottom moves between runs:

─ standing half ─────────────────────────────
  You are reviewing a pull request. …            ← the phase's instructions
  <project>    how we build and what we value    ← the project's grounding
  <standards>  what a review must cover
  <objective>  the epic this work sits under
─ changing half ─────────────────────────────
  Issue ISSUE-1 — reported token usage drops …   ← this issue
  PR #101 — 3 files, +47 −12                     ← this pull request
  <diff> …
  Verification: …                                ← what the author ran

Result: it holds. Eight passes were byte-identical (2153 bytes; standing half 1258). Assembling a second, different issue left the standing half untouched. No framework finding — the assembly was deterministic across every pass.

Nothing reusable ships: no assembly machinery, no recipe format, no store, no second phase. Three files in one goal folder, zero workspace members, zero config edits, no changes to any package. Throwaway was an acceptable outcome; the written finding is the deliverable.

The human read went red once, and that is in the log. The brief hands a reviewer a standard that asks them to check "every claim has a re-runnable evidence path", and the first version gave them nothing to check it against — no tests, no statement of what the author ran. Fixed by adding that to the world. Recorded because a gate whose red state never fires isn't a gate.

What's asked of you

Nothing here is yours to decide — this implements the approved direction in the LAB-136 spec, and its §12 records what was settled during review so it isn't reopened.

One thing is worth your awareness rather than your decision: a green here is narrower than the epic's headline claim. Eight passes seconds apart in one process rule out per-call nondeterminism. They say nothing about drift across restarts, machines, or locales. If the epic later needs cross-machine identity, that is its own issue. Please don't let this get written up as the headline claim having been met.

Parts worth reviewing closely

Every defect found on this issue has been the same shape, and it is worth stating once because it is what the check is now built to resist: an instrument that cannot see what it claims to measure, reporting fine. Five instances, each a few lines to fix — the boundary derivation, localization's ordering, contribution grading, the capture overwrite, and field placement.

  • The contribution grader. It originally graded three facts, so a dropped issue body, diff, verification statement, grounding, standards or objective left every signal green — it checked that the shape was stable, not the content. The expected set is now derived by walking the fixture (15 contributions, not 3), and the fields whose bare value can't identify them are graded on their labelled fragment (+47, −12, 3 files, #101) rather than the digits, because presence alone can't tell linesAdded from linesRemoved. Worth checking the derivation really covers what the brief claims, and that carries()'s XML-escape tolerance isn't hiding a mismatch.
  • The one-invocation assertion. Each model call overwrites the capture, so a second call would discard the first handoff and leave everything grading the last brief. Asserted rather than accumulated: this recipe is tool-free and terminal at step 0, so a second call is a regression by definition, which makes the assertion its own canary. Accumulating would have graded a case the design says can't happen.
  • localize(). The one mechanism the spec flagged as having had no adversarial pass. Building it surfaced a real defect in §8 step 7: resolveSlotValues resolves only top-level slot functions, so a function nested inside object-form context resolves later and never enters the capture. That recipe drifts the brief while the capture reports the slot perfectly stable — and the step as specified then emits a confident "the drift is in the assembly seam", a false framework finding that pauses the epic under decision 3. The fix is ordering: the constraint check runs inside the attribution, because an unresolved slot means the capture is blind, and blindness is not evidence of stability.
  • The recipe constraint is load-bearing, not stylistic — every dynamic contribution must be a top-level slot function. Enforced in the two places where it changes an outcome (inside localize, and via the nested-formatter canary), and goal.md says why, because the spec PR never merges and someone extending this recipe needs to know.
  • Whether the anti-game checks can actually fail. Ten falsification runs are tabulated at the bottom of goal.md; each was performed by breaking the check on purpose and confirming red.

Links

Spec: LAB-136 Linear document · spec PR #1321 (closes unmerged) · Issue LAB-136 · Epic LAB-68 · This PR merges.


How to review this — altitude, what's in scope, what's deliberately unsettled

How to review this. This implements an approved spec — the approach and the numbered Decisions in its §6 are already signed off by a human, so please review the code against that direction, not the direction itself. The spec lives on the issue's Linear document (the durable copy) and on the spec PR (#1321), which keeps its review history. Don't expect spec/LAB-136.md to be in this diff — the spec PR closes unmerged at approval and the spec never lands on main (BP-037).

Most valuable here: correctness on the second path (the legacy shape, the null boundary, the concurrent case, the cancel path), anything deriving an auth or routing decision from caller-controllable input, and a behaviour the tests assert around rather than on.

Already settled upstream: the approach and every §6 Decision (human-approved — if one is wrong, say so as a spec finding and it gets folded back; don't re-argue it inline), and the conventions in docs/contributing/best-practices.md.

Out of scope on this contract: local structure — which helper exists, how a function is decomposed, where a shell is hoisted. Two such suggestions were declined for that reason, not because they were wrong in the abstract.

Engineering calls — decided along the way, no product sign-off needed
  • The graded contribution set is derived from the fixture, not hand-listed. A hand-written list drifts out of agreement with what the brief promises, and the drift is invisible — it shows up as a green. Cost: the goal stops being the regression check goals/README.md describes, because the human read that would catch a content drop is a one-time verdict-log row goal:all never re-runs.
  • Ambiguous fields are graded on a labelled fragment, distinctive prose on its text. Only the expectation is tightened; the graded set stays derived, so a field added to the world and never rendered still fails. Cost of skipping it: a brief reporting transposed metrics passes every signal.
  • One invocation is asserted, not accumulated. Accumulating would grade a case the design rules out and would quietly bless a regression instead of surfacing it.
  • Localization enforces the recipe constraint inside the attribution rather than beside it. Beside it, the same nested-formatter input produced a seam verdict and a separate constraint failure — and the seam bullet came first, which is the one that triggers an escalation. Cost: a false framework finding pauses the epic.
  • Substring-overlap rejection at setup, rather than restructuring the leak comparison. Guard at the boundary, keep the comparison simple.
  • streamStep omitted from the stub. Checked, not assumed: both step methods are optional on GeneratorModel (only modelId and generate are required) and canStream is false for a tool-free, identity-less recipe, so the streaming branch is unreachable. A later recipe that adds tools or itemVisibility still cannot silently reroute to a legacy path, because canStream also requires stream or streamStep to exist.
  • Per-pass session slugs. goalSessionId's stamp is per-process, so a bare slug hands all eight passes one session; the accumulated history would make pass 2 legitimately differ from pass 1 — real drift, but not the drift under test.
  • The standing half is sliced at a hardcoded index of 1, paired with a precondition asserting exactly that. Deriving the index instead would silently adapt to a broken precondition rather than voiding the derivation.
  • Two-issue signals are skipped, not evaluated, when the precondition fails — conclusions from a void split would bury the real cause.
  • briefsAgree([]) returns false. [].every() is true, and a vacuous green is the exact failure this goal exists to prevent.
  • Restraint, the one pull I felt (§4 practice 2). containsFunction / unresolvedSlots is a check a second phase's recipe would want too, and the instinct was to put it in goals/lib. I kept it inline in run.mts. goals/README.md says to reach for a shared helper only when a third goal needs it, and this is the layer the issue exists in order not to build.
Verification
$ pnpm tsx goals/context-supply/assembles-the-same-brief-every-pass/run.mts
PASS — 8 passes of the review recipe, each over a fresh copy of the fixture world, handed the
model a byte-identical brief (2153 bytes, standing half 1258); the boundary precondition held
(exactly one leading system message); each pass called the model exactly once, so no handoff
was overwritten; all 15 contributions the brief claims to carry were present, derived by
walking the fixture rather than hand-listed and graded on their rendered placement, not bare
presence, so a value under the wrong label fails; assembling ISSUE-2 instead left the standing
half unchanged while the changing half differed, carried its own contributions and none of
ISSUE-1's identity. The comparator is proved able to fire: the same named comparator over a
grounding that appends a per-call counter returned false, and localization attributed it to
the recipe rather than the assembly seam. The recipe constraint is proved able to fire too:
for a nested context formatter the raw slot signal looks stable — the hazard — the guard sees
the unresolved function, and localization consequently refuses to blame the seam. Model n/a
throughout — the stub records what it was handed and its legacy generate() throws, so the
assembly measured is the framework's own.

$ pnpm goal:all --model-free context-supply/assembles-the-same-brief
PASS  context-supply/assembles-the-same-brief-every-pass
1/1 passed

$ cd goals && pnpm typecheck
(clean)

Falsification runs — each performed by breaking the check on purpose, confirming red, then reverting. A green from a check nobody has seen fail is decoration.

What was broken Result
Issue B assembled as issue A (the vacuous-green hole) RED — three assertions: changing halves identical, B's facts missing, A's facts leaked
A raw string added to the context slot RED — precondition saw 2 leading system messages. Also settles by execution the spec's source-read claim that a string context entry becomes an extra system message
generateStep removed from the stub RED — the run routed into legacy generate() and the guard threw
A nested-formatter drift on the eight passes, before the localization fix RED, but attributed to the seam — a false framework finding
The same drift, after the fix RED, attributed to the recipe — "ours to fix here"
The diff and the verification statement dropped from the tail RED — named by fixture path, for both issues. Every other signal stayed green, which is the point
The standards and the objective summary dropped from the grounding RED — named grounding.standards, objective.summary
A replacement world whose two issues are ISSUE-1 and ISSUE-10 RED at setup — overlapping identifiers rejected at the boundary rather than reported as a false leak
linesAdded and linesRemoved swapped in renderTail RED — named +47 and −12 as expected fragments. Fully green before labelled-fragment grading, since both digits were still in the tail
A tool added and a tool call scripted on step 0, forcing two invocations RED — "called 2 time(s), expected exactly 1. The capture is void." Before the assertion the second call would have overwritten the first brief

Coverage this does not have, stated rather than papered over: the seam verdict is reachable but has never fired on a real drift (producing one needs the framework itself made nondeterministic, i.e. editing packages/*, a non-goal); a clock read inside one tick is not reliably detected; fixture mutation is not detected either way; cross-machine/process/locale variance is out of scope.

Files
  • goals/context-supply/assembles-the-same-brief-every-pass/goal.md — the contract, the recipe constraint and why it's load-bearing, the verdict log (the human-read FAIL, the PASS, and the two review-round re-runs), and the falsification table.
  • goals/context-supply/assembles-the-same-brief-every-pass/run.mts — the recipe, the recording step-capable stub, one named comparator used by every check, the boundary precondition, the derived placement-aware contribution assertions, the constraint guard, and localization.
  • goals/context-supply/assembles-the-same-brief-every-pass/fixtures/world.json — the hand-built world. Invented identifiers only (ISSUE-1, #101), never real tracker ids (BP-006).
  • .changeset/lab-136-brief-byte-identical.md — empty frontmatter (no package bumps), with the short internal-only note the repo's other hand-named internal changesets carry.

…asses (LAB-136)

Adds a model-free goal under `goals/context-supply/` that assembles one phase's
context recipe — review — over a hand-built fixture world and checks that the
framework hands the model the same bytes on every pass, and that the brief's
standing half survives a change of issue.

Nothing reusable ships: no assembly machinery, no recipe format, no store, no
second phase. Three files in one goal folder; zero workspace members, zero
config edits, no `packages/*` changes.

Three things the check does deliberately, each of which fails GREEN if done the
obvious way instead:

- The model stub is step-capable and its legacy `generate` throws. A
  `generate()`-only stub drives the SDK-owned compatibility path production
  never takes; measuring the brief there passes while testing the wrong
  mechanism. Verified by removing `generateStep` and watching the guard fire.
- The two-issue check requires the changing halves to DIFFER and to carry their
  own fixture's facts, not just that the standing halves match. Verified by
  assembling issue A twice: three assertions go red.
- The drift canary uses a per-call counter, never the clock, and calls the same
  named comparator the real signals use.

One correction to the spec's localization step, found by building it. Capturing
slot outputs and reading "slots agree" as an assembly-seam finding is unsound:
`resolveSlotValues` resolves only top-level slot functions, so a function nested
inside object-form context resolves later and never enters the capture. That
recipe drifts the brief while the capture reports the slot stable, and the step
emits a confident false framework finding — the expensive direction, since it
pauses the epic. The fix is to run the recipe constraint INSIDE the attribution:
an unresolved slot means the capture is blind, and blindness is not evidence of
stability. Both behaviours are recorded in the verdict log.

No framework finding: the assembly seam was deterministic across every pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f
@linear-code

linear-code Bot commented Aug 18, 2026

Copy link
Copy Markdown

LAB-136

@vercel

vercel Bot commented Aug 18, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments
Project Deployment Actions Updated (UTC)
flow-state-dev-docs Ignored Ignored Aug 18, 2026 1:42am
flow-state-dev-kitchen-sink Ignored Ignored Aug 18, 2026 1:42am
flow-state-dev-ui Ignored Ignored Preview Aug 18, 2026 1:42am

Comment on lines +4 to +10
Internal only — no published package changes.

Adds one model-free goal, `goals/context-supply/assembles-the-same-brief-every-pass`, proving that a phase's context recipe hands the model a byte-identical brief on every pass and that its standing half survives a change of issue. Nothing reusable ships: no assembly machinery, no recipe format, no persistence, no second phase. Three files in one goal folder; no workspace member, no config edit, no change to any package.

The recipe is a real generator run through the real `runAction`, so what is measured is the framework's own message assembly rather than a helper written for the test. The stub is step-capable and its legacy `generate` throws, because a `generate()`-only stub drives the SDK-owned compatibility path production never takes — and would have passed green while measuring the wrong mechanism.

The assembly seam was deterministic across every pass, so there is no framework finding to file.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Populated internal-only Changeset

This change is explicitly internal-only, but the empty-package Changeset contains release-note prose, causing internal implementation details to enter the release-note pipeline. Internal-only changes should use an empty fragment.

Suggested change
Internal only — no published package changes.
Adds one model-free goal, `goals/context-supply/assembles-the-same-brief-every-pass`, proving that a phase's context recipe hands the model a byte-identical brief on every pass and that its standing half survives a change of issue. Nothing reusable ships: no assembly machinery, no recipe format, no persistence, no second phase. Three files in one goal folder; no workspace member, no config edit, no change to any package.
The recipe is a real generator run through the real `runAction`, so what is measured is the framework's own message assembly rather than a helper written for the test. The stub is step-capable and its legacy `generate` throws, because a `generate()`-only stub drives the SDK-owned compatibility path production never takes — and would have passed green while measuring the wrong mechanism.
The assembly seam was deterministic across every pass, so there is no framework finding to file.

Context Used: CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Cursor

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b0094f5771

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +318 to +320
const factsOf = (id: string): string[] => {
const issue = issueOf(WORLD, id);
return [issue.id, issue.title, `#${issue.pullRequest.number}`];

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Assert every required brief contribution

If a later recipe or framework regression deterministically drops the issue body, diff, verification, PR metrics, project grounding, standards, or objective, this goal still reports PASS: the equality and split checks remain green, while factsOf() grades only the ID, title, and PR number. The verdict log's one-time human read is not rerun by goal:all, so add fixture-derived assertions for every contribution the goal claims the reviewer receives.

Useful? React with 👍 / 👎.

if (missing.length > 0) {
failures.push(`${id}'s changing half is missing its own fixture's facts: ${missing.join(", ")}`);
}
const leaked = factsOf(other).filter((f) => tail.includes(f));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Compare held-out facts without substring collisions

When a valid replacement fixture contains overlapping facts such as ISSUE-1/ISSUE-10, #10/#101, or one title containing another, the correct longer issue tail is falsely reported as leaking the shorter issue because includes() performs a substring match. The setup guard rejects only exactly equal facts, so this breaks the stated guarantee that another valid world can be swapped in; compare structured fields or reject substring overlaps during setup.

Useful? React with 👍 / 👎.

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplify review (LAB-136 goal check)

Ran pnpm tsx goals/context-supply/assembles-the-same-brief-every-pass/run.mts — PASS in ~1.3s. Direction is sound; I would not swap this for an assembleMessages unit test or @flow-state-dev/testing's mockGenerator (legacy generate path only — your falsification table already shows that false-greens).

The execution path looks like the right minimum for "what the framework hands the model on the step path." What reads heavy is mostly presentation and attribution machinery, not a wrong architectural choice.

What I'd trim (behavior-preserving)

  1. Comment duplication — Roughly half of run.mts restates goal.md (recipe constraint, localization ordering, anti-game, decision 3). The goal doc is excellent; the runner could keep a short header (run command + pointer to goal.md) and one-line section labels. That alone is probably ~80–120 lines without weakening the proof.

  2. Session id pattern — Per-pass isolation is load-bearing. Math.random() works, but the corpus pattern is goalSessionId with a per-pass slug (e.g. goalSessionId(lab136-pass-${i})). Bare goalSessionId("lab136") would reuse one id for the whole process because RUN_STAMP is per-run, not per-pass.

  3. Micro hoist — defineFlow / recipe shell inside every assemble() is identical except the grounding closure; hoisting the static shell is a small win on 19 runAction calls (wall-clock, not correctness).

What I would not cut

  • Eight passes (signal + canary) — contract in goal.md; clock-based drift is explicitly out of scope.
  • Localization + nested-formatter guard — verdict log shows a real false "report and stop — seam" before the ordering fix. Heavy, but tied to decision 3, not speculative abstraction.
  • Two-issue standing/changing checks — closes the "assembled issue A twice" vacuous-green hole; already falsified in your table.
  • Step-capable stub with throwing generate() — correct guardrail; aligns with stepModel() in engine tests but shouldn't be imported from test internals.

Simpler approaches considered

Alternative Verdict
assembleMessages unit test (message-assembly.test.ts) Wrong surface — grades helper return, not model input
testFlow + mockGenerator Wrong path — no generateStep
Drop anti-game / localization Weaker proof; decision 3 failure mode returns
Third-party stable-stringify Doesn't buy byte identity of the messages array

Follow-ups (optional, not blocking)

  • If a second goal needs the recording stub, createMessageRecorderModel() in goals/lib/model.mts (README bar: third consumer).
  • @flow-state-dev/testing mockGenerator with generateStep would help future goals — out of scope here.
  • Promise.all on the two eight-pass loops could cut elapsed time without changing pass count.

Verdict

Comment, not approve — not because the approach is wrong, but because ~500 lines of runner with heavy goal.md duplication will be expensive to extend. A comment-trim pass (and goalSessionId alignment) would make this an easy approve without changing what the goal proves.

goal.md coverage limits (eight in-process passes ≠ production "round eight") are stated honestly — good; don't let a green here become the epic headline claim.

Open in Web View Automation 

Sent by Cursor Automation: Review For Best Approach

userId: "goal-user",
// A fresh session per pass: a shared one would accumulate history and make
// pass 2 legitimately differ from pass 1, which is not the drift we're after.
sessionId: `lab136_${Math.random().toString(16).slice(2)}`,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per-pass session isolation is load-bearing (comment above is right). For corpus consistency, consider goalSessionId(lab136-pass-${i}) in the S1/canary loops instead of Math.random() — not bare goalSessionId("lab136"), which would reuse one id for the whole process because RUN_STAMP is per-run.

user: () => record("tail", renderTail(world, issueId)),
});

const flow = defineFlow({

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The defineFlow / generator shell is identical on every assemble() except the grounding closure. Hoisting the static flow definition (and only varying grounding + issue id per call) would shave setup on 19 runAction invocations. Cosmetic for correctness; minor wall-clock win.

* Model-free (`Model: n/a`), real assembly path, out of CI. See goal.md for the
* contract; this header covers only what a reader needs to follow the code.
*
* What makes this a goal check rather than a dressed-up unit test:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplify opportunity: this header + several section blocks (~localization, recipe constraint, anti-game) largely duplicate goal.md, which is already thorough. Trimming to "contract in goal.md" + one-line section pointers would cut ~80–120 lines without weakening the proof — the runnable file should grade, not re-teach decision 3.

* epic. An unresolved slot is therefore not evidence of stability; it is evidence
* that the capture is blind, and blindness is never grounds for escalating.
*/
function localize(captures: Capture[]): Attribution {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This localization block is load-bearing (verdict log: false seam attribution before the ordering fix). I would not drop it — but localize + slotOutputsAgree + containsFunction could be one attributeDrift(captures) helper to shrink the runner without changing behavior.


Spelled that way alone, the step is unsound. The constraint check has to run **inside** the attribution, first: an unresolved slot means the capture is blind, and blindness is not evidence of stability. Before that ordering was in place, a nested recipe formatter produced a confident *"report and stop — the drift is in the assembly seam"*. It now produces *"ours to fix here, NOT a framework finding."* Both outputs are recorded in the verdict log.

**Coverage this does not have** (stated rather than papered over):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These coverage limits are exactly the kind of honesty that keeps a green goal from overselling the epic. Worth keeping prominent if you trim run.mts comments — that's the news a reader shouldn't have to dig for.

@jhoffner jhoffner left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second-look + architecture pass

Verdict: appropriately scoped, nothing to change. No bloat/YAGNI findings clear the calibration gate, and there's no architecture-improvement occasion here — this is a single throwaway goal folder (3 files + changeset), touches no packages/*, adds no workspace member, and the PR body is explicit that no reusable machinery ships. A broader refactor suggestion would itself be the kind of unrequested expansion CLAUDE.md §2/§3 argues against, so none is offered.

I did verify the two things the PR body specifically asked a reviewer to check, against the actual framework source rather than taking the claims at face value:

  1. localize()'s ordering (run.mts:434-451) is correct, and the two-stage resolution gap it depends on is real. resolveSlotValues (packages/core/src/blocks/generator.ts:685-698) only calls top-level slot functions; aggregateContextEntries (packages/core/src/blocks/context-aggregator.ts:1-42) is what resolves a function nested inside an object-form context entry, and it runs after resolveSlotValues in the pipeline (packages/core/src/blocks/internal/message-assembly.ts:172). So the capture point at run.mts:365 genuinely sits before a nested drift: () => … would get invoked — the hazard is mechanically real, not a dramatization, and running the constraint guard (unresolvedSlots) before slotOutputsAgree is the only sound order. I didn't find a case where reordering or dropping the guard would still produce a correct verdict.

  2. The generateStep-removal falsification checks out. packages/core/src/blocks/generator.ts:3055 (generation = model.generateStep !== undefined) is exactly the branch that falsification run exercises — pulling generateStep off the stub routes into legacy generate(), which the stub makes throw (run.mts:307-312) specifically to keep this from passing green while measuring the wrong path.

Also spot-checked run.mts's 497 lines against the rest of the goals/ corpus (105–1794 lines, median ~190, several single-behavior goals already at 250–450) — size is in-family, not an outlier, and the decision to keep containsFunction/unresolvedSlots inline rather than extracting to goals/lib matches the documented policy in goals/README.md ("only when a third goal needs it") rather than being a judgment call worth second-guessing.

Nothing here blocks merge.


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grok version of this simplify review (parallel pass to the other automation reviewer — treat as a second opinion, not a duplicate bot).

Verdict

Not approving. The mechanism for the real claim is excellent and should stay. The approach still carries more product-shape law and self-test surface than an eight-pass determinism proof needs. Cleanest merge would shrink that shell; if Decision 2 stays locked from the approved LAB-136 spec, say so explicitly and treat the rest of this as optional polish rather than blockers.

What went right (keep)

  • Real path, not a helper: generator → runAction → capture options.messages on generateStep. Correctly avoids calling assembleMessages directly.
  • Wrong-path guard: step-capable stub + legacy generate() throws. mockGenerator has no generateStep — reusing it would fail green on the compatibility path. Custom stub is justified.
  • Same-run localization with unresolved-first ordering: the nested-formatter hazard against resolveSlotValues / aggregateContextEntries is real; blaming the seam from a blind capture would pause the epic under Decision 3. That ordering stays.
  • Counter canary (not clock) + empty briefsAgree([]): right anti-games.
  • Home and reuse: goals/context-supply/… layout is clean; correctly uses goals/lib (loadFixture / runGoal / silentLogger); no package scaffolding, no cross-imports, empty changeset. Spec-PR concern about labs/conductor was fixed by landing here instead.
  • No meaningful perf issue for a hand-run model-free goal; no useful 3rd-party package to swap in.

Where the approach went too far

1. Decision 2 is still the main complexity dial (same finding as Grok on spec PR #1321)

Standing-half-survives-issue-change forces: role-scan precondition, hardcoded slice(0,1), two-issue assembly, fact leak suite, and a large share of goal.md / comments. That is recipe/product layout (what a phase brief should look like for caching), not “assembly is byte-identical across passes.”

A cleaner change for the epic’s reliability claim:

  1. N assemblies of one issue → same bytes
  2. On red → localize (unresolved-first → slot drift → seam)
  3. Counter canary + nested-formatter canary

Drop cross-issue standing/changing unless conductor layers that consume a standing half already exist. I’m uncertain whether product already re-ratified Decision 2 after the #1321 feedback — if yes, downgrade this to “scope bloat we accepted”; if not, this is the biggest simplify lever left.

2. Constraint / attribution covered three times

G1 always-on + localize’s first step + A3/A4 nested canary all police the same hazard. Keep localize-first and the nested canary (that’s what found the false seam). The always-on G1 when bytes already agree is optional duplication.

3. Prose weight

~500-line run.mts (~29% comment lines) and a ~9k goal.md retell the soundness story several times. Sibling inherits-parent-conversation is ~180 / ~3k. Prefer one contract in goal.md and short call-site comments; the evidence string at the end also re-narrates the thesis.

4. Small leftovers (not blockers)

  • streamStep is never exercised on this path — generateStep-only stub is enough unless the model contract requires both.
  • Canary loop at PASSES=8 is theatrical; N=2 already proves the comparator can go red.
  • Prefer goalSessionId(...) from goals/lib over lab136_${Math.random()} (sibling goal already does).

Alternative shape (if rewriting were allowed)

assemble → Capture (generateStep only; generate throws)
8× same issue → briefsAgree
on red → localize(unresolved-first)
2× counter canary → !agree && recipe
2× nested fn → !agree && raw slots “stable” && localize === recipe
print one brief for the human gate

That still proves Decision 1 + Decision 3 + behaviours that catch vacuous greens — without smuggling standing-half product law into the proof.

Disposition

COMMENT, not APPROVE. Core seam proof is the right approach; I would not ask to rewrite it onto unit assembleMessages or mockGenerator. I also would not merge while calling Decision 2 “necessary for determinism” without an explicit product call — that is where complexity was created beyond the claim.

If Decision 2 is locked and intentional, a short reply to that effect is enough for me to treat findings 2–4 as optional cleanup and reconsider approval on a follow-up.

Open in Web View Automation 

Sent by Cursor Automation: Grok Review For Best Approach

Comment on lines +368 to +401
// S3 — two DIFFERENT issues. The standing half must hold; the changing half
// must actually change and carry its own issue's facts. Without the last two,
// a recipe that silently assembled issue A twice would pass every other signal.
const a = eight[0]!;
const b = await assemble(freshWorld(), ISSUE_B, reviewGrounding);
// Gated on the precondition: with the boundary void, every conclusion below is
// drawn from a split that does not mean what it claims, and reporting them
// alongside the real cause just buries it.
if (preFail !== undefined) {
failures.push("the two-issue signals were not evaluated — the boundary precondition failed above");
} else {
if (standingHalf(a) !== standingHalf(b)) {
failures.push("the standing half changed between two different issues — the split does not hold");
}
if (changingHalf(a) === changingHalf(b)) {
failures.push(
"the changing half is IDENTICAL for two different issues — issue B was never really assembled, " +
"so the standing-half comparison above compared a brief with itself",
);
}
for (const [id, capture, other] of [
[ISSUE_A, a, ISSUE_B],
[ISSUE_B, b, ISSUE_A],
] as const) {
const tail = changingHalf(capture);
const missing = factsOf(id).filter((f) => !tail.includes(f));
if (missing.length > 0) {
failures.push(`${id}'s changing half is missing its own fixture's facts: ${missing.join(", ")}`);
}
const leaked = factsOf(other).filter((f) => tail.includes(f));
if (leaked.length > 0) {
failures.push(`${id}'s changing half carries ${other}'s facts: ${leaked.join(", ")}`);
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grok / simplify — Decision 2 surface (main overshoot).

This whole S3 block (plus the standing/changing helpers above) is most of the extra machinery beyond “eight passes, same bytes.” It encodes that a review brief should split into a standing half that survives an issue change — recipe/product layout — not that the assembly seam is deterministic.

Same concern as the Grok review on spec PR #1321. A thinner goal keeps S1 + localize + canaries and drops cross-issue standing/changing until something actually consumes a standing-half cache key.

Uncertain: if Decision 2 was deliberately re-ratified after that spec feedback, this stays as accepted scope — but then please don’t narrate it as required for the determinism claim.

Comment on lines +294 to +308
function localize(captures: Capture[]): Attribution {
const unresolved = [...new Set(captures.flatMap(unresolvedSlots))];
if (unresolved.length > 0) {
return {
verdict: "recipe",
detail:
`slot(s) [${unresolved.join(", ")}] returned an unresolved function, so their per-call values ` +
`never entered the capture. Attribution to the seam is unavailable — fix the recipe first`,
};
}
const { agree, drifted } = slotOutputsAgree(captures);
return agree
? { verdict: "seam", detail: "every slot returned the same value on every pass" }
: { verdict: "recipe", detail: `slot output varies between passes: ${drifted.join(", ")}` };
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grok / simplify — keep the ordering; trim the surrounding belt.

Unresolved-first inside localize is load-bearing (false seam → epic pause). Keep it, and keep the nested-formatter canary that proves it.

What I’d drop or collapse: the always-on G1 check when bytes already agree — it retells the same constraint localize already enforces on red. Triple coverage of one hazard is how a 150-line proof became ~500.

Comment on lines +154 to +161
async *streamStep(options: GeneratorModelCallOptions) {
capture.messages = options.messages;
yield {
type: "finish" as const,
finishReason: "stop",
fullResult: { text: "", finishReason: "stop" },
};
},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grok / simplify (low). streamStep is never hit on this non-streaming path; generateStep + throwing generate already pin the mechanism. Unless the GeneratorModel contract requires both methods present, this is dead surface in a throwaway goal.

Comment on lines +404 to +416
// A1/A2 — anti-game. A variant whose grounding appends a PER-CALL COUNTER
// (never the clock: eight in-process assemblies can land in one tick). The
// same named comparator must return false, and localization — reading the slot
// outputs captured on these same runs — must attribute it to the recipe.
let calls = 0;
const countingGrounding = (w: World): unknown => ({
...(reviewGrounding(w) as Record<string, unknown>),
drift: `call ${(calls += 1)}`,
});
const canary: Capture[] = [];
for (let i = 0; i < PASSES; i += 1) {
canary.push(await assemble(freshWorld(), ISSUE_A, countingGrounding));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grok / simplify (low / uncertain). Primary PASSES = 8 matches the claim language; fine. The canary does not need eight assemblies — a per-call counter makes briefsAgree false at N=2 (nested variant already uses 2). Shrinking only the canary loop would cut ~6 runActions without weakening the anti-game.

Comment on lines +9 to +12
**Signal:**
- **Eight passes, one issue.** Eight assemblies of the recipe, each over a fresh copy of the world, render to the same bytes under one named comparator (`briefsAgree`).
- **The precondition.** Exactly one leading `system` message and none after it — which is what makes "the leading system run *is* the standing half" exact for this recipe rather than a heuristic. If it fails, the boundary derivation is void and the standing-half signal is withdrawn, not reinterpreted.
- **Two different issues.** The standing half is byte-identical between issue A and issue B; the changing half **differs**; and each changing half carries its own fixture's facts (id, title, PR number) and **none** of the other's.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grok / simplify — contract shape.

Signal bullets 1 + anti-game (counter, wrong-path stub) carry the epic claim. Bullet 3 (two different issues / standing half) is Decision 2 and is what forces the precondition + slice + fact-leak suite in run.mts.

If the simpler approach wins, this bullet becomes “out of scope / later phase,” not a machine signal — and goal.md can shed a large share of the standing/changing prose below.

Copy link
Copy Markdown
Collaborator Author

Decision 2 is locked and intentional. Answering the question the Grok pass flagged as uncertain, since only the gate-holder can.

It was challenged at the spec gate by two reviewers on #1321, deliberately kept, and recorded in §12 as settled specifically so it wouldn't be reopened blind. The reasoning, briefly, because it's the part that wasn't written down clearly enough the first time:

Eight passes over one issue pass by construction. Nothing moved between them, so they hold whatever the recipe put wherever. They rule out per-call nondeterminism and nothing else. The cross-issue assertion is the only signal in this goal that can distinguish "the assembly is stable" from "the whole brief happens to be constant" — which means dropping it doesn't trim scope, it removes the part that can fail.

On whether it's product law smuggled into a determinism proof: that was the original objection and it was fair against the spec as written, because the spec asserted the dependency instead of arguing it. The epic's L4 claim is the standing/changing split — a stable prefix identical every run by construction plus a small changing tail. The determinism claim and the split are the same claim stated at two altitudes, and only the higher one is falsifiable. That argument was added in round 2 rather than the decision being re-asserted.

What it is not is a rule binding future recipes. Round 2 also cut that framing — §6 now records it as an observation about this recipe, not an invariant over L4, and §12 leaves open whether a later phase legitimately needs an issue-specific standing instruction.

So please treat findings 2–4 as optional cleanup, per your own disposition. Two of them are being folded anyway and are good catches: the always-on constraint check duplicating what localize already enforces on red, and the canary running eight passes where two prove the comparator fires. The streamStep question is being checked against the model contract rather than assumed either way.

One correction worth making, on size. run.mts at 497 lines was independently measured against the goals/ corpus: range 105–1794, median ~190, with several single-behaviour goals already at 250–450. Above median, inside the family — so the comment trim is worth doing for readability, but "150-line proof became ~500" overstates the anomaly.

And agreed on the thing you closed with: a green here is narrower than the epic's headline. Eight in-process passes seconds apart say nothing about restarts, machines, or locales. That's stated in goal.md and it will not be written up as the headline claim having been met.


Generated by Claude Code

…36 review round 1)

The check could report PASS while the brief lost most of its content. It graded
three facts — issue id, title, PR number — so a deterministic drop of the issue
body, the diff, the verification statement, the PR metrics, the project
grounding, the standards or the objective left every automated signal green:
eight passes still agreed, the standing/changing split still held, the canary
still fired. That made it a check on the SHAPE being stable, not the content.

The expected set is now derived by walking the fixture world rather than
hand-listed, so it cannot drift from what the brief promises: 15 contributions
graded instead of 3. Verified by dropping the diff and verification statement
(RED, named by fixture path) and separately the standards and objective summary
(RED) — with every other signal staying green, which was the point.

Also from round 1:

- Substring-overlap guard at setup. The leak check uses `includes`, so a
  replacement world holding `ISSUE-1` and `ISSUE-10` would have reported a false
  leak. Rejected at the boundary instead, keeping the held-out claim honest.
  Verified by renaming a fixture issue.
- Per-pass `goalSessionId` slugs replace `Math.random()`. The per-pass part is
  load-bearing, not cosmetic: the stamp is per-process, so a bare slug would hand
  all eight passes one session and accumulate history between them.
- Canary reduced from 8 assemblies to 2 — a per-call counter differs on the
  second. The primary loop stays at eight because that is the claim's language.
- The always-on recipe-constraint check is removed. The hazard is policed where
  it changes an outcome: `localize` consults the constraint before it may say
  "seam", and the nested-formatter canary proves that guard fires.
- `streamStep` removed from the stub. Checked rather than assumed: both step
  methods are optional on `GeneratorModel` (only `modelId` and `generate` are
  required), and this recipe is tool-free and identity-less, so `canStream` is
  false and the streaming branch is unreachable. Adding tools or `itemVisibility`
  later cannot silently reroute to a legacy path either, since `canStream` also
  requires `stream` or `streamStep` to exist.
- Comment trim where `run.mts` restated `goal.md`; the coverage-limits block
  stays in the header, where a reader meets it before the code.

Not folded: consolidating `localize` / `slotOutputsAgree` / `containsFunction`
into one helper, and hoisting the flow shell. Local structure is out of scope on
this PR's review contract, and the adjacent pull was already resisted
deliberately under the spec's §4 practice 2.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e5aa167c47

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

// a legacy path either: `canStream` also requires `stream` or `streamStep` to
// exist, and neither does.
async generateStep(options: GeneratorModelCallOptions) {
capture.messages = options.messages;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Record and validate every model invocation

If a runtime regression or retry invokes this tool-free generator more than once, each call overwrites capture.messages, discarding every earlier model handoff. An earlier malformed or pass-dependent brief can therefore be ignored while briefsAgree and the content checks grade only the final stable call, allowing the goal to pass even though the model received non-identical briefs; accumulate all invocations or assert that exactly one occurred.

Useful? React with 👍 / 👎.

Comment on lines +260 to +261
const carries = (haystack: string, leaf: string): boolean =>
haystack.includes(leaf) || haystack.includes(escapeXml(leaf));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Bind graded values to their rendered fields

Unlike the earlier identity-only leak path, the newly added S3 all-leaf grader checks only whether each scalar occurs somewhere in the half, not whether it is rendered under the correct field. With the committed fixture, swapping linesAdded and linesRemoved in renderTail leaves every leaf present and every other signal green while the brief reports incorrect metrics; a held-out fixture with overlapping values can similarly hide an omitted field. Grade field-aware rendered fragments rather than unscoped substrings.

Useful? React with 👍 / 👎.

… review round 2)

Two more instances of the failure this goal keeps finding: a check that cannot
see what it claims to measure, reporting fine.

1. The capture could lose the brief it was measuring. Each model invocation
   overwrote `capture.messages`, so a second call — from a runtime regression or
   a retry — silently discarded the first handoff and left `briefsAgree` and the
   content checks grading only the final stable call. A malformed or
   pass-dependent first brief was invisible.

   Asserted at exactly one invocation rather than accumulating. This recipe is
   tool-free and terminal at step 0, so a second call is a regression by
   definition: the assertion is both the guard and its own canary, and
   accumulating would grade a case the design says cannot happen.

2. The 15-contribution grader checked presence, not placement. It asked whether
   each scalar appeared somewhere in its half, so swapping `linesAdded` and
   `linesRemoved` left every leaf present and every signal green while the brief
   reported the wrong change size. A held-out world with two equal values hid an
   omitted field the same way.

   The ambiguous fields are now graded on their labelled fragment — `+47`, `−12`,
   `3 files`, `#101` — built from the rendered form rather than the bare value.
   The labels mirror `renderTail` and sit beside it. The graded SET is still
   derived by walking the fixture, so a field added to the world and never
   rendered still fails; only the expectation for ambiguous fields is tightened.

Both verified by breaking them: swapping the two metrics goes red naming the
expected fragments (it was fully green before), and giving the recipe a tool with
a tool call scripted on step 0 produces two invocations and goes red.

No new abstraction: +25 lines of code, no grading framework, no shared helper.
Comment share unchanged at 22%.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cd07cdcd7f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

});

const flow = defineFlow({
kind: "lab-136-review-brief",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Remove the tracking ID from runtime identifiers

Every execution creates a flow kind named after the Linear ticket, and the same tracking label also drives the session slug at line 169. This violates BP-006 in docs/contributing/best-practices/process.md, which explicitly prohibits Linear references as identifiers in code or test logic because they couple runtime artifacts to transient planning state; use stable, behavior-based names such as review-brief-goal instead.

Useful? React with 👍 / 👎.

Comment on lines +288 to +290
const expectedFragment = (leaf: Leaf): string => {
const placed = PLACED.find(([suffix]) => leaf.path.endsWith(suffix));
return placed === undefined ? leaf.text : placed[1](leaf.text);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Bind every contribution to its rendered field

Fresh evidence beyond the prior placement comment is that the new PLACED table fixes only the four PR scalars; this fallback still grades every other leaf by unscoped substring containment across the entire half. For example, swapping reviewGrounding's project and standards assignments leaves both fixture strings present, so every signal remains green while the model receives each value under the wrong XML field, contradicting the goal's “correctly placed” guarantee. Bind the remaining structured leaves to their tags or labeled regions as well.

Useful? React with 👍 / 👎.

…ds from runtime identifiers (LAB-136)

Placement grading covered four named fields; every other contribution still
graded by unscoped substring across the whole half. Swapping `project` and
`standards` in the recipe's grounding left both strings present and every signal
green while the model read each under the other's heading — the "correctly
placed" guarantee did not hold for the fields the table didn't name.

Each contribution is now graded inside its own tag's content. The tags are read
off the recipe's own context object — its keys ARE the tags the framework
renders — so this is derived from the recipe rather than declared beside it, and
a recipe that grows a fourth grounding key is covered without anyone remembering
to update the check.

The labelled fragments were KEPT rather than replaced, and that is a tested
decision rather than a cautious one. Region scoping cannot see a transposition
INSIDE a region: all four PR scalars render into the same untagged tail, so
`linesAdded` and `linesRemoved` swap without leaving it. Emptying the table and
relying on region scoping alone was run, and the transposed brief goes GREEN.
The two mechanisms cover two different hazards and neither subsumes the other;
the pair is the tested minimum. Both directions are recorded in goal.md's
falsification table, including the green, so the redundancy is not mistaken for
belt-and-braces later.

Also, BP-006: the flow kind and session slug were named after the Linear ticket,
coupling runtime artifacts to transient planning state. Renamed to
`review-brief-goal` and `review-brief-pass-N`; the per-pass structure is
unchanged, so the session-isolation property is unaffected. The only remaining
mention is the file header's provenance line, which matches the sibling goal
(`FIX-920`) and goal.md's `Issue:` field.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5dfc63933f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

] as const) {
// The changing half is one untagged region, so placement within it is what
// the labelled fragments in `PLACED` carry.
const tailMissing = missingFrom(changingContributions(id), () => changingText(capture));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Bind changing-half contributions to their rendered fields

Fresh evidence after the earlier placement feedback is that the new scoped regions cover only the standing half: every changing contribution is still searched across the entire tail here. If renderTail swaps issue.body with pullRequest.verification, both strings remain present, so S3 and every other signal stay green while the brief assigns the content to the wrong fields; bind these contributions to their labels or positional regions instead.

Useful? React with 👍 / 👎.

Region scoping landed on the standing half only; every changing contribution was
still searched across the whole tail. Swapping `issue.body` with the verification
statement left both strings present and every signal green while the brief
assigned each to the wrong field — the same hole, in the half the previous commit
did not reach.

The tail gets its OWN mechanism rather than the standing half's stretched to fit,
because the two are shaped differently and pretending otherwise would be worse
than having two: the standing half is tag-structured, so its regions come from
the recipe's context keys (which ARE the tags the framework renders); the tail is
prose with labels and has no tags, so it is sliced at the labels `renderTail`
writes — `Issue `, `PR #`, `Verification: ` — kept beside the renderer. A label
that goes missing voids the tail's scoping loudly instead of silently widening
the search back to the whole half, which is how this hole appeared.

Verified both ways: the body/verification swap goes red naming both fields, and
the metric transposition still goes red afterwards, so region scoping and the
labelled fragments compose rather than one masking the other.

Known limit, now recorded in goal.md rather than left implicit: two UNLABELLED
fields transposed inside a single region are not detected. Regions are the finest
scope the render offers, and within one region only the labelled fragments
distinguish fields — `issue.id` has `Issue `, the line counts have `+`/`−`, so
those are covered. Closing it further would mean labelling the brief for the
test's benefit, which is the tail wagging the dog.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b14dfc3556

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +359 to +360
function tailRegion(text: string, index: number): string {
const starts = TAIL_ANCHORS.map((a) => text.indexOf(a));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject fixture text that collides with tail anchors

When a valid replacement fixture contains a line beginning with PR # in an issue body, or Verification: in the body or diff, these indexOf calls select the embedded text instead of the labels emitted by renderTail. The correct recipe then fails contribution grading because its regions are sliced at the wrong offsets; for example, appending \nPR #999 is mentioned here to the first issue body makes this goal report issue.body missing. Either reject anchor collisions during setup, as is already done for identifying facts, or derive unambiguous boundaries.

Useful? React with 👍 / 👎.

claude added 2 commits August 18, 2026 01:15
…el (LAB-136)

The setup guard was half-applied. It already refuses to run on a fixture whose
two issues have overlapping identifying facts, because that would break the leak
check's own comparison. Anchor collision is the same class — fixture content
silently breaking this check's slicing — and was not covered.

`tailRegion` and `standingRegion` locate regions with `indexOf`, so a fixture
containing one of the labels sends the search to the embedded occurrence, cuts
the regions at the wrong offsets, and reports a contribution missing from a
perfectly correct recipe. Now rejected at setup in the same voice as the existing
guard, naming which label collided and where.

The current fixture is clean — verified independently, with the instrument
sanity-checked against a string known to be present — so the standing PASS and
the verdict log are unaffected. This closes the door, it does not withdraw a
result.

The guarded list is deliberately asymmetric, and the asymmetry is now backed by a
run rather than a source read: grounding and objective values reach the standing
half through `renderTaggedContext`, which escapes `<` and `>`, so a value
containing `<project>` cannot collide with a tag. Confirmed by putting exactly
that into a grounding value and watching the goal pass rather than be rejected —
the guard does not over-reject a valid world. The prompt and the whole tail are
written verbatim, so those are what is checked.

Proved able to fire in both branches, then reverted: embedding "PR #" and
"Verification: " in an issue body is rejected naming both, and embedding
"<project>" in the unescaped prompt is rejected naming that.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f
The run behind this row was made on a clean tree at e380252, the commit that
implements the anchor-collision guard — the row cites that SHA rather than the
branch base, so a later reader can check out exactly what was run.

Adds the three falsification rows behind the guard, including the one that
PASSES: a grounding value containing `<project>` is XML-escaped and cannot
collide, so it is correctly not rejected. That row is the evidence for the
guard's asymmetric list, and it is recorded as a pass on purpose — a guard that
over-rejects valid worlds would break the held-out property this goal depends on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5367e513c1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +151 to +152
context: () => record("grounding", grounding(world)),
user: () => record("tail", renderTail(world, issueId)),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Move structured brief data into context slots

This recipe puts the complete issue, PR, diff, and verification dump in user while returning all grounding from one context function. BP-017 in docs/contributing/best-practices/generators.md requires structured data in context: { tagName: fn } and reserves user for a short trailing instruction; because this goal becomes the reusable regression record for brief assembly, it currently validates and documents a prompt shape the repository explicitly prohibits. Split these contributions into tagged context slots and keep user short, or surface the grounding conflict instead of canonizing this exception.

AGENTS.md reference: AGENTS.md:L11-L15

Useful? React with 👍 / 👎.

Comment on lines +306 to +307
const carries = (haystack: string, fragment: string): boolean =>
haystack.includes(fragment) || haystack.includes(escapeXml(fragment));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Distinguish contributions instead of matching substrings

Fresh evidence after the earlier overlap fix is that only cross-issue identity collisions are guarded; arbitrary leaves in the same region can still overlap. For example, with a valid replacement fixture whose body repeats its title, removing the title from renderTail still passes because this includes() finds the title inside the body, so the goal reports all contributions present despite the regression. Reject all same-region leaf overlaps during setup or grade structured, field-specific fragments.

Useful? React with 👍 / 👎.

Comment on lines +486 to +487
TAIL_ANCHORS.map((a) => a.replace(/^\n/, ""))
.filter((a) => l.text.includes(a))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve newlines when checking tail-anchor collisions

The new setup guard strips the leading newline from "\nPR #" and "\nVerification: ", so a valid fixture that merely mentions PR #123 inline in a title or body is rejected even though tailRegion() searches for the newline-prefixed anchor and that inline text cannot alter its boundary. This makes held-out fixture substitution fail for realistic content; test collisions against the exact anchors rather than their stripped substrings.

Useful? React with 👍 / 👎.

claude added 2 commits August 18, 2026 01:31
…136)

The guard added in e380252 stripped the leading newline off each anchor before
testing, but `tailRegion` searches for the newline-prefixed form. A fixture that
merely mentioned "PR #123" mid-sentence was rejected as invalid even though that
text cannot move a region boundary. Reproduced before fixing.

Tested against the RENDERED tail rather than per-leaf, because neither per-leaf
test is exact and the collision is a property of the joined text:

  - matching the bare `PR #` rejects an inline mention that cannot move a
    boundary (the over-rejection above);
  - matching the newline-prefixed anchor against a leaf misses a body that
    STARTS with it and picks the newline up from the join.

Verified that second case rather than reasoning about it: a body beginning
"PR #999 tracked the earlier attempt." does not contain "\nPR #" as a leaf, yet
produces a genuine duplicate in the rendered tail. An anchor occurring exactly
once is the precise condition, and `renderTail` is a pure function of the
fixture, so setup can just ask it.

Verified in both directions: the inline mention now passes, and both a body line
starting with an anchor and one containing it are rejected.

The asymmetry between escaped and verbatim renders is unchanged and still holds.

Also adds a one-line note above the recipe recording why BP-017 does not apply:
it governs production recipes, where structured material belongs in `context`;
this recipe must populate all three slots and cover both the escaped and verbatim
paths, so routing the tail through `context` would make the check test less.
Written down so the next reviewer does not re-raise it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f
…ing (LAB-136)

The run behind this row was made on a clean tree at 5a44782, the commit that
fixes the anchor guard's over-rejection — the row cites that SHA, not the branch
base.

Also records the limit being tracked on FIX-1183 rather than folded: grading is
substring matching over rendered text, and every guard in this check exists to
protect that approach against arbitrary substituted content. The fourth guard was
introduced by the third one's fix, which is the signal that the approach is the
hole rather than any individual gap. Written into the coverage list so a reader
meets it as a known ceiling, not as an oversight.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2262c9e452

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

}

// S2 — the precondition the standing/changing derivation depends on.
const preFail = preconditionFailure(eight[0]!.messages);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Check the held-out brief's message boundary

When an assembly regression emits a later system message only for issue B, this checks only issue A's message shape; S3/S4 then slice B after index 1, treating the extra system message as changing content while contribution, standing-half, and leak checks can still pass. The goal therefore reports PASS despite its explicit precondition of one leading system message and none afterward being false for one of the two briefs; validate b.messages before evaluating the two-issue signals.

Useful? React with 👍 / 👎.

// and `renderTail` is a pure function of the fixture, so setup can just ask.
...WORLD.issues.flatMap((issue) => {
const rendered = renderTail(WORLD, issue.id);
return TAIL_ANCHORS.filter((a) => rendered.indexOf(a) !== rendered.lastIndexOf(a)).map(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Allow harmless later Issue anchors

Fresh evidence after the earlier newline-anchor fix is that the new exact-once guard also applies to "Issue ". Because renderTail() always emits that anchor at offset 0 and tailRegion() uses indexOf, a second occurrence in a valid title or body—for example, “The previous Issue was closed”—cannot move any boundary, yet this setup check rejects the held-out fixture before exercising the correct recipe. This breaks the goal's promised fixture swappability; reject only occurrences capable of preceding or moving the canonical boundary.

Useful? React with 👍 / 👎.

claude added 2 commits August 18, 2026 01:41
Removes the guard added in e380252 and corrected in 5a44782, in full. This
backs out a decision, not a mistake in carrying it out.

The guard rejected a fixture whose content contained a region label, on the
argument that the setup validation was otherwise half-applied. Two attempts
over-rejected valid worlds in opposite directions:

  - matching a bare `PR #` refused a body that merely mentions "PR #123"
    mid-sentence, which cannot move a boundary;
  - requiring each anchor to occur exactly once refused a title or body
    containing the word "Issue", even though `renderTail` always emits that
    anchor at offset 0 and the slicing takes the first match, so no boundary can
    move there either.

Both breaks damaged the fixture swappability the guard existed to protect, and
it was never protecting anything real: the fixture in the repo has no
collisions, so every recorded verdict was sound throughout. When completing a
mechanism costs more than the mechanism is worth, the answer is to back it out
rather than keep paying.

Verified the revert restores what the guard broke: a world whose body mentions
`PR #123` and whose text contains "Issue" now passes, and the goal is green on
the real fixture.

The limit itself is not lost — it is recorded in goal.md's coverage list
alongside the two others of the same family (bare-substring matching, and the
message-boundary precondition being asserted for issue A only). All three are one
problem: grading by substring over rendered text cannot reliably localize a
contribution, and each guard bolted on is a patch on that. Tracked as its own
issue; the fix is structured, field-specific comparison.

Stale falsification rows for the removed guard are deleted rather than left
describing behaviour that no longer exists, and replaced by the one that now
matters: those worlds must PASS.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f
The run behind this row was made on a clean tree at 172d472, the commit that
backs out the anchor-collision guard — the row cites that SHA, not the branch
base.

The two verdict rows for the guard's own commits stay: they record runs that
genuinely happened, and a verdict log is a history rather than a current-state
summary. This row says plainly that what they describe has since been removed, so
a later reader is not misled into thinking the guard is still there.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVcrTezeLMBQRVHRg2QX2f

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3d9cf7e042

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +418 to +419
const first = JSON.stringify(captures[0]?.slots[name]);
return captures.some((c) => JSON.stringify(c.slots[name]) !== first);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Treat missing slot invocations as seam failures

If the framework skips a top-level slot callback on one pass, that capture has no corresponding slots property; this comparison treats the missing value as ordinary slot-output drift, so localize() reports recipe even though assembly caused the divergence. That sends the goal down its “fix here, do not file a framework finding” path for exactly the seam regression it is meant to localize; validate that every expected slot was invoked in every capture and classify an absent invocation as seam drift or void attribution before comparing values.

Useful? React with 👍 / 👎.

Comment on lines +226 to +227
if (messages.slice(lead).some((m) => roleOf(m) === "system")) {
return `a system message appears after the leading run (roles: ${messages.map(roleOf).join(", ")})`;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Require the changing message to retain the user role

If message assembly regresses and wraps the user tail as an assistant or tool message, this precondition still passes because it rejects only later system messages. The contribution checks discard roles through textOf(), and stable role corruption also passes the equality checks, so the goal can report PASS while the model interprets the issue, diff, and verification as prior conversation rather than the request; for this history-free recipe, assert the expected system, user role sequence.

Useful? React with 👍 / 👎.

@jhoffner
jhoffner merged commit 07ee630 into main Aug 19, 2026
16 of 17 checks passed
@jhoffner
jhoffner deleted the fix/LAB-136 branch August 19, 2026 11:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants